Method and apparatus for correcting camera tilt distortion in 
panoramic images 



FIELD OF THE INVENTION 

5 The present invention relates generally to panoramic photography. 

BACKGROUND OF THE INVENTION 

Panoramic cameras, that is, cameras capable of taking photographs with 

unusually wide horizontal fields of view, have existed for some time. This disclosure 
10 is concerned with panoramic photographs taken with rotational and swing-lens 

panoramic cameras. 

A rotational panoramic camera rotates the camera body during the taking of a 

photograph, and may be capable of taking photographs with horizontal viewing angles 

of 360 degrees or more. The film inside the camera also moves in synchronization 
1 5 with the camera rotation. These cameras are sometimes called "Cirkut-type" cameras, 

after an early commercial model. A well-known contemporary camera of this type is 

the Roundshot, made by Seitz Phototechnik AG of Switzerland. 

A swing-lens panoramic camera rotates the lens and an exposure slit but not 

the camera body. The film is held stationary in a curved shape and the image is 
20 painted onto the film as the exposure slit passes by it. This type of camera usually can 

take photographs with horizontal viewing angles just less than 180 degrees. Two 

well-known camera models that operate on this principle are the Widelux and Horizon 

cameras. 

Digital panoramic cameras are also available. A digital panoramic camera 
25 rotates the entire camera while repeatedly reading image data from a row or set of 
rows of electronic light sensors behind the lens. 
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Most panoramic cameras have fixed focal length lenses that are fixed in 
position in relation to the film, and thus have a limited vertical field of view that is not 
adjustable. Some cameras provide a lens shift capability wherein the lens optical axis 
may be moved relative to the film. This allows the vertical field of view of the 
5 camera to be shifted up and down, but does not increase the vertical field of view. 
However, many panoramic cameras do not provide a lens shift capability. 

Often, a panoramic camera is held level, with its rotational axis substantially 
vertical during the taking of a photograph so that the horizon in the photograph 
appears straight and level and to minimize distortion of objects in the photograph. If 
10 the camera is tilted so that its rotational axis is at an angle to vertical, objectionable 
camera-tilt distortion may be introduced into the photograph. Many camera models 
have level-indicating devices built into the camera bodies to aid the photographer in 
leveling the camera. 

A panoramic camera may also be used with its rotational axis substantially 
1 5 horizontal in order to achieve an unusually large vertical field of view, or at some 
other angle with respect to vertical. Generally, however, these cameras are used to 
achieve a large horizontal field of view and are therefore used with their rotational 
axes substantially vertical. 

This desire to keep the camera level and lack of adjustability may limit the 
20 photographer's ability to capture the desired elements of a scene. For example, the 
photographer may wish to photograph an entire building, the height of which would 
fit within the camera's vertical field of view, but be unable to include the top of the 
building in the photograph because of the requirement to keep the camera level to 
avoid distortion. 
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The photographer may be able to take the photograph from a more remote 
vantage point, thus allowing the camera's vertical field of view to encompass the 
entire building. However, this results in an undesirable loss of resolution of the 
subject in the resulting photograph, and a more remote vantage point may not be 
5 available. 

In this situation the photographer has historically been faced with the choice of 
photographing the available vertical field of view which is not the desired field of 
view, tilting the camera to encompass the desired subject and accepting the resulting 
distortion, or using a more remote vantage point if one is available and accepting the 
1 0 resulting loss of subject resolution. 

There is a need for a method of allowing the photographer to tilt a rotational or 
swing-lens lens panoramic camera, take a photograph that includes the resulting 
distortion, and recover an undistorted representation of the photograph later. 

15 SUMMARY OF THE INVENTION 

A method removes camera tilt distortion from a digital representation of a 
photograph taken with a rotational or swing-lens panoramic camera. 

BRIEF DESCRIPTION OF THE DRAWINGS 

20 Figure 1 is a diagram illustrating the operation of a rotational or swing-lens 

panoramic camera. 

Figure 2 shows an example photograph taken with a level rotational or swing- 
lens panoramic camera. 

Figure 3 illustrates the path of exposure covered when a panoramic camera is 

25 tilted. 
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Figure 4 shows an example photograph taken with a tilted rotational or swing- 
lens panoramic camera. 

Figure 5 illustrates a coordinate system used to describe an example 
embodiment of the invention. 

Figure 6 illustrates a tilted panoramic camera in the coordinate system of 
Figure 5. 

Figure 7 shows the photograph of Figure 4 after the camera tilt distortion has 
been removed by an example embodiment of the invention. 

Figures 8A and 8B depict a flow chart describing the steps performed in an 
example embodiment of the inventions. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 

Figure 1 schematically illustrates the operation of a rotational or swing-lens 
panoramic camera (100). Lens (101) is assumed to be a pinhole lens, which is 
1 & 15 modeled, as a point in space. Using a point lens model simplifies the description of 
the invention and introduces no appreciable error. An actual panoramic camera may 
use a lens including one or more reflective or refractive elements. An exposure slit 
(102) limits the horizontal field of view projected onto the image surface (103) by the 
lens (101) at any one time. The image surface (103) is conceptually depicted as a 
20 cylinder of film. This shape may be an accurate depiction for a swing lens camera. A 
Cirkut-type camera may hold its film in a different shape but present it to the lens by 
virtue of the camera and film motion so that the effect is as if the film were a cylinder. 
A digital panoramic camera effectively replaces the film with an electronic array light 
sensor. 
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As the camera (100) or lens (101) and slit (102) rotate, a complete image is 
sweepingly projected onto the cylindrical image surface (103). A camera body (not 
shown) excludes stray light not entering through the lens from hitting the cylindrical 
image surface (103). The vertical field of view (104) is determined by the focal 
5 length of the lens (101) and the height of the image surface (103). Trees (105-1 1 1) 
are partially within the field of view of the camera (100), and are placed on a level 
plane. The camera vertical field of view (104) is seen to include the base of each tree, 
but not the top. In Figure 1, the camera (100) is held level, so that its axis of rotation 

4f is vertical. Direction arrow (112) indicates an arbitrary direction of rotation for the 

^ 10 camera or lens and slit. Tracks (113) and (1 14) indicate the conceptual path taken by 
the edges of the camera field of view (104) at a particular distance from the camera. 

i=§ The camera field of view (104) extends to infinity away from the camera, but is 

q shown as truncated in Figure 1 . 

q Cylindrical image surface (103) may be a cylinder of film, or may be the 

ry 

15 conceptual effective shape of film presented to the camera lens (101) by virtue of 
motion of the camera and film. Alternatively, image surface (103) may be the 
conceptual path of an electronic light sensor in a digital camera. 

Figure la illustrates a coordinate system for describing the motion and 
parameters of the camera (100). Coordinate axes X, Y, and Z define an orthogonal 

20 coordinate space. The X-Y plane is horizontal and the Z axis is vertical. The Y-Z 
plane (104) serves as a meridional plane of the camera (100). The Z axis is the 
rotational axis of the camera when the camera is held level as illustrated. 

Figure 2 shows an example photograph that may have been taken with the 
camera (100) as shown in Figure 1. The tops of the equal-sized trees (105-111) do 

25 not show in the photograph because they extended beyond the vertical field of view 
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(104) of the camera (100). The trees (105-111) appear uniform and undistorted 
because the camera (100) was held level. The horizon (201) in the photograph is 
straight and level. 

In Figure 3, the photographer has tilted the camera (100) so that its axis of 
5 rotation is no longer vertical, causing its vertical field of view (104) to encompass the 
top of the center tree (108) in the photograph. As shown, the rotation and tilt of the 
camera (100) cause the field of view to sweep in an arc across the trees (105-111). 
Conceptual track (113) passes above the center tree (108), but falls below the tops of 
the end trees (105, 111). 
10 Figure 3a shows the cylindrical image surface (103) tilted and placed in the 

coordinate system of Figure la. Vector (301) depicts the rotational axis of the 
camera, and is at an angle with respect to the Z axis, indicating that the camera (100) 
is not level. Plane (302) is perpendicular to vector (301) and is an equatorial plane of 
the camera. 

15 Figure 4 shows an example photograph that may have been taken with the 

camera (100) as shown in Figure 3. The center tree (108) is now framed completely 
in the vertical coverage of the photograph. However, the horizon (201) in the 
photograph is no longer straight or level, and the trees (105, 106, 107, 109, 1 10, 1 1 1) 
away from the center of the photograph appear to "lean" into the center of the 

20 photograph in general relation to their distance from the center of the photograph. 
This distortion is induced by the tilt of the axis of rotation of the camera (100), and is 
actually part of a sinusoidal pattern that would be apparent if a complete 360-degree 
photograph were illustrated. Point (401) indicates the zenith point of the path traced 
by the lens optical axis. This point serves an origin point of the distorted image for 

25 later calculations. 



H. P. docket number 10017016 



Page 6 of 23 



Measurements M x and M y indicate the coordinates in pixels relative to the 
location of origin (401) of an arbitrary example pixel (402) used to demonstrate an 
example embodiment of the invention. M x and M y indicate the location of the pixel in 
a digital representation of the photograph. Both M x and M y for the example pixel 
5 shown are negative numbers. For other locations in the image, M x or M y or both may 
be positive. 

The general approach used in an example embodiment of the invention is to 
H; populate the finished rectified digital image by computing, for each pixel location in 

=3 the rectified image, the corresponding pixel location in the original distorted digital 

4f 10 image representation and copying, or "pulling" pixel data from the distorted digital 
J 1 ? image into the rectified digital image. It is also possible to start with pixels in the 

distorted digital image, calculate their corresponding locations in the rectified image, 
= and place, or "push" the pixel data into the rectified image at the corresponding 

% location. However, the "pulling" method chosen avoids difficulties that may arise due 

15 to nonuniform pixel spacing introduced by the transformation between the two 
images, simplifies interpolation, and also allows arbitrary scaling of the image. 

Figure 5 further illustrates a several parameters used to describe the operation 
of an untilted rotational or swing-lens panoramic camera (100). The coordinate origin 
serves as a model of a pinhole lens (101). A cylindrical image surface (103) has a 
20 radius equal to the lens focal length f, measured in millimeters. The cylindrical image 
surface (103) is centered on the Z axis. A location on the image surface (103) 
corresponds to a pixel (501) in a digital representation of the image. A second pixel 
(502) located at the point in the image corresponding to the intersection of the image 
surface (103) with the Y axis serves as an origin for pixel measurements in the digital 
25 image. A vector (503) defines a viewing direction corresponding to pixel (501), and 



H. P. docket number 10017016 



Page 7 of 23 



Ul 



is coincident with a line connecting the lens (101) at the coordinate origin with pixel 

(501) . 

The location in the digital image of pixel (501) in relation to the origin pixel 

(502) is given by pixel coordinates N x and N y . N x is the number of pixels that pixel 
(501) is displaced from the origin pixel (502) angularly around the camera axis. N y is 
the number of pixels that pixel (501) is displaced from the origin pixel in a direction 
parallel to the camera rotational axis. The resolution of the digital image is given by 
Ri and is measured in pixels per millimeter. The direction cosines of the viewing 
direction vector (503) are given by: 

4 = i 1 sill (^r) C 1 ) 

1 =cos(^f-) (2) 



^{N y IJR x ) 2 +1 fa' 



These equations (1) through (3) map a pixel (501) in the desired rectified 
image to camera viewing directions described by viewing direction vector (503). One 
15 of skill in the art will recognize that equations (1) through (3) may be modified to 
accommodate a digital image whose X-direction resolution differs from its Y- 
direction resolution. 

Figure 6 illustrates a tilted panoramic camera in the same coordinate system. 
Unit vector (301), having its origin at the coordinate origin and having direction 
20 cosines (0, B p , C p ), is coincident with the axis of the tilted cylindrical image surface 
(103) and is the rotational axis of the camera (100). Rotational axis (301) forms an 
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angle a with the Z axis, defining an angle of tilt of the camera (100). By virtue of this 
tilt: 



Unit vector (301) also defines plane (302) perpendicular to vector (301), 
which is an equatorial plane of the camera (100). The intersection of the tilted image 
surface (103) and the equatorial plane (302) forms a circular arc. 

Unit vector (503), having its origin at the coordinate origin and having 
direction cosines (Ai, B\, Q) represents a viewing direction of the camera. An 
extension of unit vector (503) intersects the tilted cylindrical image surface (103) at 
point (x, y, z), representing where objects in the viewing direction indicated by unit 
vector (503) will be imaged on the tilted image surface (103), and corresponding to 
pixel (402) in the example image of Figure 4. 

The Y-Z plane (104) serves as a central meridional plane of the camera (100). 
The point (x, y, z) may be thought of as being angularly displaced by an angle d about 
the camera rotational axis (301) from the camera's central meridional plane (104). 
Point (x, y, z) may also be thought of as displaced by a distance h from the camera 
equatorial plane (302) along a direction parallel to the camera rotational axis (301). 

The coordinates of point (x, y, z) are given by: 



B p = -sin(a) 



(4) 



C p = cos(a) 



(5) 



x = 



-4/ 



(6) 





zM. 



(7) 
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, ~ CJ (8) 

A /(c p 5 / -5 ; ,c / ) 2 +4 2 

The angular displacement angle 6>and displacement distance h are given by: 

6> = ^-cos- L ( ; C P y- B P Z ) ifx>0 (9a) 

^{C p y-B p zf +(C p x) 2 +(B p x) 2 

fl = cos -' ( _^ C ?y- B p z ) - n ifx <o (9b) 

^{C p y-B p z) 2 +{C p xf +(B p x) 2 

h = B p y + C p z (10) 
Point (602) defines the point at which the lens optical axis intersects tilted 
cylindrical image surface (103) when the camera is viewing in the direction parallel to 
meridional plane (302). Point (602) also represents the point on the tilted cylindrical 
image surface (103) where objects in the direction of zenith point (401) are imaged. 
A pixel at point (602) in a digital representation of the image serves as the origin for 
pixel measurements in the distorted image. 

The location in the distorted digital image of a pixel in relation to the origin 
pixel (602) is given by pixel coordinates M x and M y . M x is the number of pixels that 
the pixel is displaced from the origin pixel (602) horizontally in the image (or 
angularly around the camera axis). M y is the number of pixels that the pixel is 
displaced vertically from the origin pixel (or in a direction parallel to the camera 
axis). M x and M y locate point (x, y, z) in pixel coordinates corresponding to physical 
measurements 6 and h respectively. Where R 2 is the resolution of the distorted digital 
image measured in pixels per millimeter, M x and M y are given by: 

M x =f9R 2 (11) 
M y =hR 2 (12) 
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One of skill in the art will recognize that equations (11) and (12) may be 
modified to accommodate a digital image whose X-direction resolution differs from 
its Y-direction resolution. 

Equations (5) through (12) map a viewing direction (503) to a location in a 
digital representation of a distorted photograph taken with a tilted camera. Image data 
may be copied or extracted from this pixel location and placed in a rectified digital 
image at location N x , N y . M x and M y need not be integers. The data may be copied or 
extracted using nearest neighbor, bilinear, bicubic, or another kind of interpolation. 

Equations (1) through (12), applied in order to a location N x , N y in the desired 
rectified digital image, compute the location M x , M y in the original distorted image 
from where pixel data should be copied. 

The camera optical structure comprises the lens (101) focal length f, the image 
surface (103) height and placement, and the rotation of the camera or the lens and 
exposure slit (102). 

Figure 7 shows the photograph produced by applying an example embodiment 
of the invention to the photograph of Figure 4. The trees (105-1 11) in the photograph 
are uniform and undistorted. The horizon (201) in the photograph is straight and 
level. The center tree (108) is completely present in the photograph. The frame of the 
photograph (701) is now curved to reflect the true path traversed by the camera field 
of view (104) during the taking of the photograph. Measurements N x and N y indicate 
the coordinates in pixels of the arbitrary example pixel (501) used to demonstrate an 
example embodiment of the invention. N x and N y indicate the location of the pixel in 
a digital representation of the photograph. Both N x and N y for the example pixel are 
negative numbers. 
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Figures 8A and 8B illustrate the steps involved in removing camera tilt 
distortion from a panoramic photograph according to an example embodiment of the 
invention. These steps may be typically performed with the aid of a processor. 
In step 802, the focal length / of the camera lens (101) is identified. 
In step 804, the tilt angle a of the camera (100) is identified. This serves to 
define unit vector (601) having direction cosines {0, B p , C p ). 

In step 806, a digital image representation, or digital image, of the photograph 
containing camera tilt distortion is obtained. Such a digital image may be obtained 
directly if the camera is a digital panoramic camera, or may be obtained by scanning a 
negative, positive, print, or display produced using a camera that records its 
photographs on film. 

In step 808, the resolution R 2 of the digital image containing camera tilt 
distortion is identified. 

In step 810, the zenith point is identified in the digital image containing 
camera tilt distortion. The zenith point is the point in the image corresponding to the 
viewing direction at which the camera lens optical axis reaches its maximum angle a 
from horizontal. 

In step 812, an output digital image is allocated. This step may be entirely 
conceptual. It is not necessary that memroy storage sufficient to hold the entire 
digital image be reserved. 

In step 814, the resolution Ri of the rectified output digital image is identified. 

Steps 816 through 828 are performed for each pixel in the rectified output 
digital image. 

In step 816, a pixel location is selected in the rectified output digital image and 
its location is specified by coordinates N x and N y 
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In step 818, the pixel location N x , N y is mapped to a viewing direction 
described by unit vector (503) having direction cosines (Ai, B h Cj). 

In step 820, the intersection point (x, y, z) is computed. This point is the 
intersection of a line coincident with viewing direction unit vector (503) and the tilted 
5 cylindrical image surface (103). 

In step 822, the angular displacement #and linear displacement h of the image 
pixel at (x, y, z) from the origin point (602) are computed. 

In step 824, the angular displacement 0 and linear displacement h are mapped 
to a pixel location in the distorted digital image representation. The pixel location is 
1 0 specified by coordinates M x and M y 

In step 826, pixel data is copied or extracted from location M x , M y in the 
distorted digital image representation, by nearest neighbor, bilinear, bicubic, or other 
interpolation means. 

In step 828, the copied or extracted pixel data is placed in location N x , N y of 
1 5 the rectified output image. 

The method described above will typically be implemented with the aid of a 
stored program running on a computer or data processing system. The data 
processing system may be a personal computer comprising a central processing unit 
such as a microprocessor, primary memory such as random access memory (RAM) or 
20 read-only memory (ROM) or the like, secondary memory such as a magnetic disk 
drive or magnetic tape drive or the like, and one or more input/output devices such as 
video display terminals or printers or network connections or the like. The data 
processing system may also be in a camera. The program may be distributed on 
floppy disk, CD ROM, memory card, via a network, or by other means or media, or 
25 may be installed in the camera itself. During the running of the program, all or part of 
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a digital representation of a photograph including camera tilt distortion, as depicted in 
Figure 4, may be stored in the computer's memory. The entire digital image 
including distortion may be stored on a disk drive associated with the computer, on a 
floppy disk, on a magnetic tape, on a memory card, at a remote location accessible 
over a network, or by some other means or media. All or part of a rectified digital 
image, as depicted in Figure 7, may also be stored in the computer's memory. The 
entire rectified digital image, at the completion of the distortion removal or as the 
distortion removal progresses, may also be stored on a disk drive associated with the 
computer, on a floppy disk, on a magnetic tape, at a remote location accessible over a 
network, or by some other means or media. The program may display the rectified 
digital image. The program may also print the rectified digital image. 

The foregoing description of the present invention has been presented for 
purposes of illustration and description. It is not intended to be exhaustive or to limit 
the invention to the precise form disclosed, and other modifications and variations 
may be possible in light of the above teachings. For example, various dimensions 
used in the characterization of the camera and digital image representations may be 
expressed in different units of measurement that used in the above disclosure. The 
embodiment was chosen and described in order to best explain the principles of the 
invention and its practical application to thereby enable others skilled in the art to best 
utilize the invention in various embodiments and various modifications as are suited 
to the particular use contemplated. It is intended that the appended claims be 
construed to include other alternative embodiments of the invention except insofar as 
limited by the prior art. 
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